Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unified Test runs in Docker #2082

Merged
merged 18 commits into from
Oct 20, 2023
Merged

Conversation

AlvinSchiller
Copy link
Collaborator

The test executions are currently running sequentially and build the docker image multiple times. Also the cache is not used, as the code base is included as one of the first layer, so a change in the repo triggers an entire rebuild of the image.
Furthermore are the files mostly duplicates and hard to maintain.

Included Changes:

Dockerfiles

  • all Dockerfiles have been merged into a single file (based on debian)
  • Debian Version Name can dynamically be passed as buildargs --build-args="DEBIAN_VERSION_NAME=xxx"
  • the used platform is defined by the caller --platform=linux/arm/v7
  • the command order has been changed to maximize cache hits (least to most frequently changing).
    • installation of repo lists and packages (takes the most build time if not used from cache)
    • Add Default User (pi) and depending on target also an alternative User (hans) (selectable with --user=xxx from caller)
    • Copy files from repository
    • Update packages to deal with long living caches
  • Different targets have been added to support different use cases, e.g build for automatic tests (test-code and test-update), build for user tests (code)
  • for automatic test builds only the installscript and the testscripts are copied into the image, instead of the whole codebase

Workflows

  • all "docker" workflow-files have been merged into two files. A trigger workflow and a reusable subworkflow
  • tested debian versions are defined in a matrix in the trigger workflow. Each version entry will call the reusable subworkflow with all needed parameter, e.g. user and scripts to test. These run in parallel.
  • the subworkflow performes following steps:
    • checkout and setup steps
    • Build the base docker image for the specific debian version (see Dockerfile target 'test-code'). Use cache to speed up creation.
    • Build an updated docker image based on the base image (see Dockerfile target 'test-update'). Dont use Cache to deal will long living caches of the base image.
    • Run test execution for users and testscripts in a matrix. Each will run in parallel and uses the previously build 'updated docker image'
  • the workflow has now a concurreny setting which will cancel previous runs if a new one is triggered (new commits mostly invalided previous results, so why proceed). This is based on the current branch ref (differentiates between branches and pull requests -> https://docs.github.com/en/actions/learn-github-actions/contexts#github-context 'github_ref')
  • the cache is also differentiated on branch ref, so the first run on a branch / pull request will always create a new base image.

Additional changes

  • "ci "Readme updated
  • replaced deprecated apt-key usage
  • fix installscript checkout only from 'https://github.com/MiczFlor/RPi-Jukebox-RFID.git#develop'. GIT_BRANCH and GIT_URL are now set based on current Branch.
  • fix 'debconf: unable to initialize frontend: Dialog' warning during installation (export DEBIAN_FRONTEND=noninteractive not working with sudo apt-get ... )
  • determine current user in installscript with whoami rather then $USER to have higher compatibility with containers
  • pythonpackage_future3.yml has been removed (not relevant in 2.x branches)

Information about the caching

  • each repository has 10GB of cache space. if the limit is exceeded the oldest cached items will be deleted.
  • caches will be deleted after not been used for 7 days (scheduled tasks once per week keeping cache alive?)

removed now obsolete "*_altuser" Dockerfile
makes sure that the installation script uses the current repo/branch
only include the scripts that are needed.
Otherwise every code change will invalidate the container cache.
changed internal path of container and test scripts
changed thestructure to use the cache as much as possible.
repo url -> packages -> user -> scripts
with ref instead of ref_name pull request will get a different cache
to be compatible with containers
get rid of warning "debconf: unable to initialize frontend: Dialog"
export DEBIAN_FRONTEND=noninteractive does not work with sudo
@s-martin
Copy link
Collaborator

Cool enhancement, thanks!

Anything missing from your side otherwise I would merge it?

I would also manually adapt my Bookworm PR on the new implementation (still struggling with the enforcement of PEP668 on bookworm).

@AlvinSchiller
Copy link
Collaborator Author

Anything missing from your side otherwise I would merge it?

Should be good for now. I have some further ideas but that would be another PR.

@s-martin s-martin merged commit 597d6ee into MiczFlor:develop Oct 20, 2023
23 checks passed
@AlvinSchiller AlvinSchiller mentioned this pull request Nov 7, 2023
7 tasks
AlvinSchiller added a commit to AlvinSchiller/RPi-Jukebox-RFID that referenced this pull request Dec 21, 2023
* removed obsolete v3.x workflow files

* set user and group as build args to test altuser

removed now obsolete "*_altuser" Dockerfile

* set GIT_BRANCH and GIT_URL as build args

makes sure that the installation script uses the current repo/branch

* use correct variables for refname resolving

* harmonized workflow and Dockerfiles

* optimize test docker performance. copy changed

only include the scripts that are needed.
Otherwise every code change will invalidate the container cache.
changed internal path of container and test scripts

* optimize test docker performance. refactored

changed thestructure to use the cache as much as possible.
repo url -> packages -> user -> scripts

* changed cache key

with ref instead of ref_name pull request will get a different cache

* get username with whoami

to be compatible with containers

* implemented different strategies for uniformity

* set noninteractive on debconf

get rid of warning "debconf: unable to initialize frontend: Dialog"
export DEBIAN_FRONTEND=noninteractive does not work with sudo

* unified workflow for debian. delete obsolete files

* fix branch name on pr

* refactored into subworkflow

* fixed apt-key deprecation

* unified Dockerfile. renamed workflow files

* fix repo name on pullrequest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants